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NETWORK SURVEILLANCE statistical profile and a long-terra statistical profile indicates 

suspicious network activity. A response may include altering 

REFERENCE TO GOVERNMENT FUNDING analysis of network packets and/or severing a communica- 
tion channel. A response may include transmitting an evenfl 

This invention was made with Government support under ^ ^^^o^d to a network monitor, such as hicrarchicaUy higher I 

Contract Number F30602-96-C-0294 awarded by DARPA. network monitor and/or a network monitor that receivesj 

The Government has certain rights in this invention. event reconis from muUiple network monitors. 

R PFFR Fisrrp TO APPFMDTX network entity may be a gateway, a router, or a proxy 

Kb^hRhNLb 1 U AFl-bNUlA ^^^^^ network enUty may instead be a virtual private 

A microfiche appendix is included as pari of the specifi- iq network entity (e.g., node), 

cation. The microfiche appendix includes material subject to In general, in another aspect, a method of network sur- 

copyright protection. The copyright owner does not object to veillance includes monitoring network packets handled by a 

the reproduction of the microfiche appendix, as it appears in network entity and building a long-term and multiple short- 

the Patent and Trademark Office patent file or records, but terni statistical profiles of the network packets. A comparPt 

othenvise reserves all copyright rights. This application 15 son of one of the mulUple short-term statistical profiles w^ 

contains Microfiche Appendix containing ten (10) slides and [^e long-term staUstical profile is used to determine whether 

956 frames => v / difference between the short-term statistical profiles and i 

^ the long-term statistical profile indicates suspicious network I 

BACKGROUND ^^^^^^y. 

Embodiments may include one or more of the following. 

The invention relates to computer networks. 20 -j^^ multiple short-term statistical profiles may monitor 

Computer networks offer users ease and efficiency in different anonymous FTP sessions. Building multiple short- 
exchanging information. Networks tend to include conglom- term statistical profiles may include deinterleaving packets 
erates of integrated commercial and custom-made to identify a short-term statistical profile, 
components, interoperating and sharing information at in general, in another aspect, a computer program 
increasing levels of demand and capacity. Such varying product, disposed on a computer readable medium, includes 
networks manage a growing list of needs including instructions for causing a processor to receive network^ 
transportation, commerce, energy management, packets handled by a network entity and to build at least one 
communications, and defense. long-term and at least one short-term statistical profile from 

Unfortunately, the very interoperability and sophisticated at least one measure of the network packets that monitors' 

integration of technology that make networks such valuable data transfers, errors, or network connections. The instruc- 

assets also make them vulnerable to attack, and make tions compare a short-term and a long-term statistical profile 

dependence on networks a potential liability. Numerous to determine whether the difference between the short-term 

examples of planned network attacks, such as the Internet statistical profile and the long-term statistical profile indi- 

worm, have shown how ioterconnectivity can be used to cates suspicious network activity. 

spread harmful program code. Accidental outages such as In general, in another aspect, a method of network sur- 
the 1980 ARPAnet collapse and the 1990 AT&T collapse veillance includes receiving packets at a virtual private 
illustrate how seemingly localized triggering events can network entity and statistically analyzing the received pack- 
have globally disastrous effects on widely distributed sys- ets to determine whether the packets indicate suspicious 
tems. In addition, organized groups have performed mali- network activity. The packets may or may not be decrypted 
cious and coordinated attacks against various online targets. before statistical analysis. 

Advantages may include one or more of the following. 
Using long-term and a short-term statistical profiles from 
In general, in one aspect, a method of network surveil- measures that monitor data transfers, errors, or network 
lance includes receiving network packets (e.g., TCP/IP 45 connections protects network components from intrusion, 
packets) handled by a network entity and building at least As long-term profiles represent "normal" activity, abnormal 
one long-term and at least one short-term statistical profile activity may be detected without requiring an administrator 
from at least one measure of the network packets that to catalog each possible attack upon a network. Additionally, 
monitors data transfers, errors, or network connections. A the ability to deinterleave packets to create multiple short- 
comparison of at least one long-term and at least one 5Q term profiles for comparison against a long-term profile 
short-term statistical profile is used to determine whether the enables the system to detect abnormal behavior that may be 
difference between the short-term statistical profile and the statistically ameliorated if only a single short-term profile 
long-term statistical profile indicates suspicious network was created. 

activity. The scheme of communication network monitors also 

Embodiments may include one or more of the following 55 protects networks from more global attacks. For example, an 

features. The measure may monitor data transfers by moni- attack made upon one network entity may cause other 
toring network packet data transfer commands, data transfer! entities to be alerted. Further, a monitor that collects event 
errors, and/or monitoring network packet data transfer vol-] reports from different monitors may correlate activity to 

ume. The measure may monitor network connections by identify attacks causing disturbances in more than one 

monitoring network connection requests, network connec- go network entity. 

tion denials, and/or a correlation of network connections Additionally, statistical analysis of packets handled by a 
requests and network connection denials. The measure may virtual private network enable detection of suspicious net- 
monitor errors by monitoring error codes included in a work activity despite virtual private network security tech- 
network packet such as privilege error codes and/or error niques such as encryption of the network packets, 
codes indicating a reason a packet was rejected. 65 Other features and advantages will become apparent from 
The method may also include responding based on the the following description, including the drawings, and from 
determining whether the difference between a short-term the claims. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a diagram of network monitors deployed in an 
enterprise. 

FIG. 2 is a diagram of a network monitor that monitors an ^ 
event stream. 

FIG. 3 is a diagram of a resource object thai configures the 
network monitor of FIG. 2. 

FIG. 4 is a flowchart illustrating network surveillance. 

FIG. 5 is a flowchart illustrating multiple short-term jq 
statistical profiles for comparison against a single long-term 
statistical profile, 

FIG. 6 is a diagram of a computer platform suitable for 
deployment of a network monitor. 

DETAILED DESCRIPTION 

Referring to FIG. 1, an enterprise 10 includes different 
domains Vla-llc. Each domain lla-\2c includes one or 
more computers offering local and network services that 
provide an interface for requests internal and external to the 
domain 12a-12c. Network services include features com- 
mon to many network operating systems such as mail, 
HTTP, FTP, remote login, network file systems, finger, 
Kerberos, and SNMP. Some domains I2a-12c may share 
trust relationships with other domains (either peer-to-peer or 
hierarchical). Alternatively, domains \2a~\2c may operate 
in complete mistrust of all others, providing outgoing con- 
nections only or severely restricting incoming connections. 
Users may be local to a single domain or may possess 
accounts on multiple domains that allow them to freely 
establish connections throughout the enterprise 10. 

As shown, the enterprise 10 includes dynamically 
deployed network monitors 16a-16/ that analyze and 
respond to network activity and can interoperate to form an 
analysis hierarchy. The analysis hierarchy provides a frame- 35 
work for the recognition of more global threats to interdo- 
main connectivity, including coordinated attempts to infil- 
trate or destroy connectivity across an entire network 
enterprise 10. The hierarchy includes service monitors 
16fl-16c, domain monitors 16if-16e, and enterprise moni- 
tors 16/ 

Service monitors 16a-l 6c provide local real-time analy- 
sis of network packets (e.g., TCP/IP packets) handled by a 
network entity 14a-14c. Network entities include gateways, 
routers, firewalls, or proxy servers, A network entity may 45 
also be part of a virtual private network. A virtual private 
network (VPN) is constructed by using public wires to 
connect nodes. For example, a network could use the 
Internet as the medium for transporting data and use encryp- 
tion and other security mechanisms to ensure that only 50 
authorized users access the network and that the data cannot 
be intercepted. A monitor 16fl-16/can analyze packets both 
before and after decryption by a node of the virtual private 
network. 

Information gathered by a service monitor 16fl-l 6c can 55 
be disseminated to other monitors 16^-16/, for example, via 
a subscription-based communication scheme. In a 
subscription-based scheme cfient monitors subscribe to 
receive analysis reports produced by server monitors. As a 
monitor 16a-16/ produces analysis reports, the monitor 60 
16i2-16/ disseminates these reports asynchronously to sub- 
scribers. Through subscription, monitors 16^-16/ distrib- 
uted throughout a large network are able to efficiently 
disseminate reports of malicious activity without requiring 
the overhead of synchronous polling. 65 

Domain monitors perform surveillance over all 

or part of a domain 12a-12c. Domain monitors 16rf-16e 
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correlate intrusion reports disseminated by individual ser- 
vice monitors 16fl-16c, providing a domain- wide perspec- 
tive of activity (or patterns of activity). In addition to domain 
surveUlance, domain monitors 16fl-16c can reconfigure 
system parameters, interface with other monitors beyond a 
domain, and report threats against a domain Ma-llc to 
administrators. Domain monitors 16i/-16e can subscribe to 
service monitors 16fl-16c, Where mutual trust among 
domains lla-llc exists, domain monitors 16d-16e may 
establish peer relationships with one another. Peer-to-peer 
subscription allows domain monitors 16d-16e to share 
analysis reports produced in other domains Xla-llc, 
Domain monitors 16d-16e may use such reports to dynami- 
cally sensitize their local service monitors I6a-16c to mali- 
cious activity found to be occurring outside a domain 
Vla-llc. Domain monitors 16(^-16e may also operate 
within an enterprise hierarchy where they disseminate analy- 
sis reports to enterprise monitors 16/ for global correlation. 

Enterprise monitors 16/ conelate activity reports pro- 
duced across the set of monitored domains 12a-12c. Enter- 
prise 10 surveillance may be used where domains lla-llc 
are interconnected under the control of a single organization, 
such as a large privately owned WAN (Wide Area Network). 
The enterprise 10, however, need not be stable in its con- 
figuration or centrally administered. For example, the enter- 
prise 10 may exist as an emergent entity through new 
interconnections of domains 12a~Mc. Enterprise 10 surveil- 
lance is very similar to domain lla-llc surveillance: an 
enterprise monitor 16/ subscribes to various domain moni- 
tors 16i/-16e, just as the domain monitors 16d-16e sub- 
scribed to various service monitors 16fl-16c. The enterprise 
monitor 16/ (or monitors, as it would be important to avoid 
centralizing any analysis) focuses on network- wide threats 
such as Internet worm-like attacks, attacks repeated against 
common network services across domains, or coordinated 
attacks from multiple domains against a single domain. As 
an enterprise monitor 16/ recognizes commonalities in intru- 
sion reports across domains (e.g., the spreading of a worm 
or a mail system attack repeated throughout the enterprise 
10), the monitor 16/ can help domains lla-llc counter the 
attack and can sensitize other domains lla-llc to such 
attacks before they are affected. Through correlation and 
sharing of analysis reports, reports of problems found by one 
monitor 16a-16/may propagate to other monitors 16a-16/ 
throughout the network. Interdomain event analysis is vital 
to addressing more global, information attacks against the 
entire enterprise 10. 

Referring to FIG. 2, each monitor 16 includes one or more 
analysis engines 22, 24. These engines 22, 24 can be 
dynamically added, deleted, and modified as necessary. In 
the dual-analysis configuration shown, a monitor 16 instan- 
tiation includes a signature analysis engine 22 and a statis- 
tical profihng engine 24. In general, a monitor 16 may 
include additional analysis engines that may implement 
other forms of analysis. A monitor 16 also includes a 
resolver20 that implements a response poUcy and a resource 
object 32 that configures the monitor 16, The monitors 16 
incorporate an application programmers' interface (API) 
that enhances encapsulation of monitor functions and eases 
integration of third -party intrxision-detection tools 28, 30. 

Each monitor 16 can analyze event records that form an 
event stream. The event stream may be derived from a 
variety of sources such as TCP/IP network packet contents 
or event records containing analysis reports disseminated by 
other monitors. For example, an event record can be formed 
from data included in the header and data segment of a 
network packet. The volume of packets transmitted and 
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received, however, dictates careful assessment of ways to 
select and organize network packet information into event 
record streams. 

Selection of packets can be based on different criteria. 
Streams of event records can be derived from discarded 
traffic (i.e., packets not allowed through the gateway because 
they violate filtering rules), pass-through traffic (i.e., packets 
allowed into the internal network from external sources), 
packets having a common protocol (e.g., all I CMP (Internet 
Control Message Protocol) packets that reach the gateway), 
packets involving network connection management (e.g., 
SYN, RESET, ACK, [window resize]), and packets targeting 
ports to which an administrator has not assigned any net- 
work service and that also remain unblocked by the firewall. 
Event streams may also be based on packet source addresses 
(e.g., packets whose source addresses match well-known 
external sites such as satellite offices or have raised suspi- 
cion from other monitoring efforts) or destination addresses 
(e.g., packets whose destination addresses match a given 
internal host or workstation). Selection can also implement 
application-layer monitoring (e.g., packets targeting a par- 
ticular network service or application). Event records can 
also be produced from other sources of network packet 
information such as report logs produced by network enti- 
ties. Event streams can be of very fine granularity. For 
example, a different stream might be derived for commands 
received from * different commercial web-browsers since 
each web-browser produces different characteristic network 
activity. 

A monitor 16 can also construct interval summary event 
records, which contain accumulated network traffic statistics 
(e.g., number of packets and number of kilobytes 
transferred). These event records are constructed at the end 
of each interval (e.g., once per N seconds). Event records are 
forwarded to the analysis engines 22, 24 for analysis. 

The profile engine 22 can use a wide range of multivariate 
statistical measures to profile network activity indicated by 
an event stream. A statistical score represents how closely 
currently observed usage corresponds to the established 
pattems of usage. The profiler engine 22 separates profile 
management and the mathematical algorithms used to assess 
the anomaly of events. The profile engine 22 may use a 
statistical analysis technique described in A. Valdes and D. 
Anderson, "Statistical Methods for Computer Usage 
Anomaly Detection Using NIDES", Proceedings of the 
Third International Workshop on Rough Sets and Soft 
Computing, January 1995, which is incorporated by refer- 
ence in its entirety. Such an engine 22 can profile network 
activity via one or more variables called measures. Measures 
can be categorized into four classes: categorical, continuous, 
intensity, and event distribution measures. 

Categorical measures assume values from a discrete, 
nonordered set of possibilities. Examples of categorical 
measures include network source and destination addresses, 
commands (e.g., commands that control data transfer and 
manage network connections), protocols, error codes (e.g., 
privilege violations, malformed service requests, and mal- 
formed packet codes), and port identifiers. The profiler 
engine 22 can build empirical distributions of the category 
values encountered, even if the list of possible values is 
open-ended. The engine 22 can have mechanisms for "aging 
out" categories whose long-term probabiUties drop below a 
threshold. 

Continuous measures assume values from a continuous or 
ordinal set. Examples include inter-event time (e.g., differ- 
ence in time stamps between consecutive events from the 
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same stream), counting measures such as the number of 
errors of a particular type observed in the recent past, the 
volume of data transfers over a period of time, and network 
traffic measures (number of packets and number of 
kilobytes). The profiler engine 22 treats continuous mea- 
sures by first allocating bins appropriate to the range of 
values of the underlying measure, and then tracking the 
frequency of observation of each value range. In this way, 
multi-modal distributions are accommodated and much of 
the computational machinery used for categorical measures 
is shared. Continuous measures are useful not only for 
intrusion detection, but also to support the monitoring of the 
health and status of the network from the perspective of 
connectivity and throughput. For example, a measure of 
traffic volume maintained can detect an abnormal loss in the 
data rate of received packets when this volume falls outside 
historical norms. This sudden drop can be specific both to 
the network entity being monitored and to the time of day 
(e.g., the average sustained traffic rate for a major network 
artery is much different at 11:00 a.m. than at midnight). 

Intensity measures reflect the intensity of the event stream 
(e.g., number of ICMP packets) over specified time intervals 
(e.g., 1 minute, 10 minutes, and 1 hour). Intensity measures 
are particularly suited for detecting flooding attacks, while 
also providing insight into other anomalies. 

Event distribution measures are meta-measures that 
describes how other measures in the profile are affected by 
each event. For example, an "Is" command in an FTP 
session affects the directory measure, but does not affect 
measures related to file transfer. This measure is not inter- 
esting for all event streams. For example, all network-traffic 
event records affect the same measures (number of packets 
and kilobytes) defined for that event stream, so the event 
distribution does not change. On the other hand, event 
distribution measures are useful in correlative analysis per- 
formed by a monitor 16fl-16/ that receives reports from 
other monitors 16a-16/ 

The system maintains and updates a description of behav- 
ior with respect to these measure types in an updated profile. 
The profile is subdivided into short-term and long-term 
profiles. The short-term profile accumulates values between 
updates, and exponentially ages (e.g., weighs data based on 
how long ago the data was collected) values for comparison 
to the long-term profile. As a consequence of the aging 
mechanism, the short-term profile characterizes recent 
activity, where "recent" is determined by a dynamically 
configurable aging parameters. At update time (typically, a 
time of low system activity), the update function folds the 
short-term values observed since the last update into the 
long-term profile, and the short-term profile is cleared. The 
long-term profile is itself slowly aged to adapt to changes in 
subject activity. Anomaly scoring compares related 
attributes in the short-term profile against the long-term 
profile. As all evaluations are done against empirical 
distributions, no assumptions of parametric distributions are 
made, and multi-modal and categorical distributions are 
accommodated. Furthermore, the algorithms require no a 
priori knowledge of intrusive or exceptional activity. 

The statistical algorithm adjusts a short-term profile for 
the measure values observed in the event record. The 
distribution of recently observed values is compared against 
the long-term profile, and a distance between the two is 
obtained. The difference is compared to a historically adap- 
tive deviation. The empirical distribution of this deviation is 
transformed to obtain a score for the event. Anomalous 
events are those whose scores exceed a historically adaptive 
score threshold based on the empirical score distribution. 
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This nonparametric approach handles all measure types and 
makes no assumptions on the modality of the distribution for 
continuous measures. 

Profiles are provided to the computational engine as 
classes defined in the resource object 32. The mathematical 5 
functions for anomaly scoring, profile maintenance, and 
updating do not require knowledge of the data being ana- 
lyzed beyond what is encoded in the profile class. Event 
collection interoperability supports translation of the event 
stream to the profile and measure classes. At that point, 10 
analysis for different types of monitored entities is math- 
ematically similar. This approach imparts great flexibility to 
the analysis in that fading memory constants, update 
frequency, measure type, and so on are tailored to the 
network entity being monitored. 

The measure types described above can be used individu- 
ally or in combination to detect network packet attributes 
characteristic of intrusion. Such characteristics include large 
data transfers (e.g., moving or downloading files), an 
increase in errors (e.g., an increase in privilege violations or ^ 
network packet rejections), network connection activity, and 
abnormal changes in network volume. 

As shown, the monitor 16 also includes a signature engine 
24. The signature engine 24 maps an event stream against 
abstract representations of event sequences that are known 
to indicate undesirable activity. Signature-analysis objec- 
tives depend on which layer in the hierarchical analysis 
scheme the signature engine operates. Service monitor 
16fl-16c signature engines 24 attempt to monitor for 
attempts to penetrate or interfere with the domain's opera- 
tion. The signature engine scans the event stream for events 
that represent attempted exploitations of known attacks 
against the service, or other activity that stands alone as 
warranting a response from the monitor. Above the service ^5 
layer, signature engines 24 scan the aggregate of intrusion 
reports from service monitors in an attempt to detect more 
global coordinated attack scenarios or scenarios that exploit 
interdependencies among network services. Layering signa- 
ture engine analysis enables the engines 24 to avoid mis- 
guided searches along incorrect signature paths in addition 
to distributing the signature analysis. 

A signature engines 24 can detect, for example, address 
spoofing, tunneling, source routing, SATAN attacks, and 
abuse of I CMP messages ("Redirect" and "Destination 45 
Unreachable" messages in particular). Threshold analysis is 
a rudimentary, inexpensive signature analysis technique that 
records the occurrence of specific events and, as the name 
implies, detects when the number of occurrences of that 
event surpasses a reasonable count. For example, monitors 50 
can encode thresholds to monitor activity such as the num- 
ber of fingers, pings, or failed login requests to accounts 
such as guest, demo, visitor, anonymous FTP, or employees 
who have departed the company. 

Signature engine 24 can also examine the data portion of 55 
packets in search of a variety of transactions that indicate 
suspicious, if not malicious, intentions by an external client. 
The signature engine 24, for example, can parse FTP traffic 
traveling through the firewall or router for unwanted trans- 
fers of configuration or specific system data, or anonymous 60 
requests to access non -public portions of the directory 
structure. Similarly, a monitor can analyze anonymous FTP 
sessions to ensure that the file retrievals and uploads/ 
modifications are limited to specific directories. 
Additionally, signature analysts capability can extend to 65 
session analyses of complex and dangerous, but highly 
useful, services like HTTP or Gopher 
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Signature analysis can also scan trafiBc directed at unused 
ports (i.e., ports to which the administrator has not assigned 
a network service). Here, packet parsing can be used to study 
network traffic after some threshold volume of traffic, 
directed at an unused port, has been exceeded. A signature 
engine 24 can also employ a knowledge base of known 
telltale packets that are indicative of well-known network- 
service protocol traffic (e.g., FTP, Telnet, SMTP, HTTP). 
The signature engine 24 then determines whether the 
unknown port traffic matches any known packet sets. Such 
comparisons could lead to the discovery of network services 
that have been installed without an administrator's knowl- 
edge. 

The analysis engines 22, 24 receive large volumes of 
events and produce smaller volumes of intrusion or suspi- 
cion reports that are then fed to the resolver 20. The resolve r 
20 is an expert system that receives the intrusion and 
suspicion reports produced by the analysis engines 22, 24 
and reports produced externally by other analysis engines to 
which it subscribes. Based on these reports, the resolver 20 
invokes responses. Because the volume of intrusion and 
suspicion reports is lower than the volume of events 
received by the analysis engines 22, 24, the resolver 20 can 
afford the more sophisticated demands of configuration 
maintenance and managing the response handling and exter- 
nal interfaces necessary for monitor operation. Furthermore, 
the resolver 20 adds to extensibility by providing the sub- 
scription interface through which third-party analysis tools 
28, 30 can interact and participate in the hierarchical analy- 
sis scheme. 

Upon its initialization, the resolver 20 initiates authenti- 
cation and subscription sessions with those monitors 
16fl-16/ whose identities appear in the monitor's 16 
subscription-list (46 FIG. 3). The resolver 20 also handles all 
incoming requests by subscribers, which must authenticate 
themselves to the resolver 20. Once a subscription session is 
established with a subscriber monitor, the resolver 20 acts as 
the primary interface through which configuration requests 
are received and intrusion reports are disseminated. 

Thus, resolvers 20 can request and receive reports from 
other resolvers at lower layers in the analysis hierarchy. The 
resolver 20 forwards analysis reports received from sub- 
scribees to the analysis engines 22, 24. This tiered collection 
and correlation of analysis results allows monitors 16a-16/ 
to represent and profile global malicious or anomalous 
activity that is not visible locally. 

In addition to external-interface responsibilities, the 
resolver 20 operates as a fully functional decision engine, 
capable of invoking real-time response measures in response 
to malicious or anomalous activity reports produced by the 
analysis engines. The resolver 20 also operates as the center 
of intramonitor communication. As the analysis engines 22, 
24 build intrusion and suspicion reports, they propagate 
these reports to the resolver 20 for further correlation, 
response, and dissemination to other monitors 16c-16/. The 
resolver 20 can also submit runtime configuration requests 
to the analysis engines 22, 24, for example, to increase or 
decrease the scope of analyses (e.g., enable or disable 
additional signature rules) based on various operating met- 
rics. These configuration requests could be made as a result 
of encountering other intrusion reports from other subscrib- 
ers. For example, a report produced by a service monitor 
16a-16c in one domain could be propagated to an enterprise 
monitor 16/, which in turn sensitizes service monitors in 
other domains to the same activity. 

The resolver 20 also operates as the interface mechanism 
between administrators and the monitor 16. From the per- 
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spective of a resolver 20, the administrator interface is to its parent enterprise monitor 16/. The enterprise monitor 
simply a subscribing service to which the resolver 20 may 16/ operates as a client to one or more domain monitors 
submit reports and receive configuration requests. An 16d—16e, allowing them to conclate and model enterprise- 
administrative interface tool can dynamically subscribe and wide activity from the domain-layer results. Domain moni- 
unsubscribe to any of the deployed resolvers 20, as well as 5 tors I6d-16e operate as servers to the enterprise monitors 
submit configuration requests and asynchronous probes as ^^f' ^ ^^^^^^ ^ service monitors 16fl-16c deployed 

throughout their domain 12a-12c. This message scheme can 
„ * . . , operate substantially the same i f correlation were to continue 
The monitors 16a-l6f incorporate a bidirectional mes- ^, (^^^ ^ „f abstraction beyond enterprise 10 analysis, 
saging system that uses a standard intetface specification for ,„tramonitor and intermonitor programming interfaces 
communication withm and between monitor elements and . * n tt_ • . c u uj- 
, , , . - . , r -n • 1 are substantially the same. These interfaces can be subdi- 
external modules. Using this interface specification, third- „. , . . «f ;„t^.««or.t^««. ;«;t;oi 
J I -in * -* I- video mto five categones ot interoperation: channel initial- 
party modules 28, 30 can communicate with monitors. For . ^. . T^. . 1 u ■ 4* J 
^ ^ , , ' ji^o u * * J* ization and tcrmmation, channel synchronization, dynamic 
example, third-party modiU^ 28 can submit event records to ,„„fi ,bing, and report/event dissemina- 
the analysis engines 22, 24 for processing. Additionally, ti„„. ^clients are responsible for initiating and terminaUng 
third-party modules 30 may aUosubnii and receive anai^ 15 ^^^^^^ ^^^^ ^.^ ^^^^^^ ^^^^^ responsibll 

results via the resolver's 20 external interfaces. Thus, third- ^ «f *™rc 

J 1 -so i/\ • * *u c for managing channel synchronization in the event or errors 

party modules 28, 30 can incorporate the results from ^ ^ . ^ - a ce '\ a i 

^ . . .1. Ml a: ^ . . • m message sequencing or penods of failed or slow response 

monitors into other surveillance efforts or contribute their ^ ^„«fi™fH•,^„c^ nw^^i^ r«,t. ,io^ oT^KT*,;t 

- , . ^/rr T *t .u > (i-e., I m alive connrmations). Clients may also submit 

result to other monitors 16ii-16/. Lastly, the monitor s 16 \ . c . \ o „ i 

, , , , . -^^ . . , 1. 1 J on dynamic configuration requests to servers. For example, an 

internal API allows third-party analysis engines to be hnked ^0 ^ ^ ^^^^^ ^M^^M^r, 

directly mto the momlor boundary. ^^^^^ ^ ^^^^^ -^^ ^j^^^^^ semantics. Clients may also 

The message system operates under an asynchronous pj-obe servers for report summaries or additional event 

communication model for handling results dissemination information. Lastly, servers may send clients intrusion/ 

andprocessingthatisgenerically referred to as subscription- suspicion reports in response to client probes or in an 

based message passing. Component interoperation is client/ asynchronous dissemination mode. 

server-based, where a client module may subscribe to ^^^^^ ^^^^ ^y^^ message system fi-amework 

receive event data or analysis results from servers. Once a .^^^^^^^ specification of a transport mechanism used to 

subscription request is accepted by the server, the server estabUsh a given communication channel between monitors 

module forwards events or analysis results to the chent x^^-Uf or possibly between a monitor 16a-16/ and a 

automatically as data becomes avadable, and may dynami- thi^d-party security module. AU implementation dependen- 

cally reconfigure itself as requested by the client s control ^^^^ ^^jj^.^ message system framework are addressed by 

requests. This asynchronous model reduces the need for pluggable transport modules. Transport modules are specific 
chent probes and acknowledgments. participating intrusion-detection modules, their 

The interface supports an implementation-neutral com- respective hosts, and potentially to the network — should the 

munication framework that separates the programmer's modules require cross-platform interoperation. Instantiating 

interface specification and the issues of message transport. a monitor 16a-16/may involve incorporation of the neces- 

The interface specification embodies no assumptions about sary transport module(s) (for both internal and external 

implementation languages, host platform, or a network. The communication). 

transport layer is architecturally isolated from the internals i^e transport modules that handle intramonitor commu- 
of the monitors so that transport modules may be readily nication may be different from the transport modules that 
introduced and replaced as protocols and security require- intermonitor communication. This allows the intra- 
menU are negotiated between module developers. The inter- monitor transport modules to address security and reliability 
face specification involves the definition of the messages i^su^g differently than how the intermonitor transport mod- 
that the various intrusion-detection modules must convey to ^les address security and reliability. While intramonitor 
one another and how these messages should be processed. communication may more commonly involve interprocess 
The message structure and content are specified in a com- communication within a single host, intermonitor commu- 
pletely implementation-neutral context. nication will most commonly involve cross-platform net- 
Both intramonitor and intermonitor communication worked interoperation. For example, the intramonitor trans- 
employ identical subscription-based client-server models. 50 port mechanisms may employ unnamed pipes which 
With respect to intermonitor communication, the resolver 20 provides a kernel-enforced private interprocess communi- 
operates as a client to the analysis engines, and the analysis cation channel between the monitor 16 components (this 
engines 22, 24 operate as clients to the event filters. Through assumes a process hierarchy within the monitor 16 
the internal message system, the resolver 20 submits con- architecture). The monitor's 16 external transport, however, 
figuration requests to the analysis engines 22, 24, and 55 will more likely export data through untrusted network 
receives from the analysis engines 22, 24 their analysis connections and thus require more extensive security man- 
results. The analysis engines 22, 24 operate as servers agement. To ensure the security and integrity of the message 
providing the resolver 20 with intrusion or suspicion reports exchange, the external transport may employ public/private 
either asynchronously or upon request. Similarly, the analy- key authentication protocols and session key exchange, 
sis engines 22, 24 are responsible for establishing and Using this same interface, third-party analysis tools may 
maintaining a communication link with an event collection authenticate and exchange analysis results and configuration 
method (or event filter) and prompting the reconfiguration of information in a well-defined, secure manner, 
the collection method's filtering semantics when necessary. The pluggable transport permits flexibility in negotiating 
Intermonitor communication also operates using the security features and protocol usage with third parties, 
subscription -based hierarchy. A domain monitor 16^^-1 6e 65 Incorporation of a commercially available network manage- 
subscribes to the analysis results produced by service moni- ment system can deliver monitoring results relating to 
to rsl 6 6c, and then prop agates its own analytical reports security, reliability, availability, performance, and other 
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attributes. The network management system may in turn stances under which the method should be invoked. These 

subscribe to monitor produced results in order to influence metrics include a threshold metric that corresponds to the 

network reconfiguration. measure values and scores produced by the profiler engine 

All monitors (service, domain, and enterprise) 16a-16/ 22 and severity metrics that correspond to subsets of the 

use the same monitor code-base. However, monitors may 5 associated attack sequences defined within the resource 

include different resource objects 32 having different con- object 32. 

figuration data and methods. This reusable software archi- Countcrmeasures range from very passive responses, such 

tecture can reduce implementation and maintenance efforts. as report dissemination to other monitors 16a-l6f or 

Customizing and dynamically configuring a monitor 16 thus administrators, to highly aggressive actions, such as scvcr- 

becomes a question of building and/or modifying the ^0 jng a communication channel or the reconfiguration of 

resource object 32. logging facilities within network components (e.g., routers. 

Referring to FIG. 3, the resource object 32 contains the firewaUs, network services, audit daemons). An active 

operating parameters for each of the monitor^s 16 compo- response may invoke handlers that validate the integrity of 

nents as well as the analysis semantics (e.g., the profiler network services or other assets to ensure that privileged 

engine's 22 measure and category definition, or the signa- network services have not been subverted. Monitors 

ture engine's 24 penetration rule-base) necessary to process 16fl-16/may invoke probes in an attempt to gather as much 

an event stream. After defining a resource object 32 to counterintelligence about the source of suspicious traffic by 

implement a particular set of analyses on an event stream, using features such as traceroute or finger, 

the resource object 32 may be reused by other monitors 16 The resource object 32 may include a subscription list 46 

deployed to analyze equivalent event streams. For example, '^^ that includes information necessary for establishing 

the resource object 32 for a domain's router may be reused subscription-based communication sessions, which may 

as other monitors 16 are deployed for other routers in a include network address information and public keys used 

domain 12a-12c. A library of resource objects 32 provides by the monitor to authenticate potential clients and servers, 

prefabricated resource objects 32 for commonly available Hie subscription list 46 enables transmission or reception of 

network entities. ^ messages that report malicious or anomalous activity 

The resource object 32 provides a pluggable configuration between monitors. The most obvious examples where rela- 

module for tuning the generic monitor code-base to a tionships are important involve interdependencies among 

specific event stream. The resource object 32 includes network services that make local policy decisions. For 

configurable event structures 34, analysis unit configuration example, the interdependencies between access checks per- 

3Sa-3Sn, engine configuration 40fl-40rt, resolver configu- formed during network file system mounting and the IP 

ration 42, decision unit configuration 44, subscription list mapping of the DNS service. An unexpected mount moni- 

data 46, and response methods 48. tored by the network file system service may be responded 

Cbnfigurable event strucmres 34 define the structure of differently if the DNS monitor informs the network file 
event records and analysis result records. The monitor 35 systemmonitor of suspicious updates to the mount request- 
code -base maintains no internal dependence on the content ^ mapping. 

or format of any given event stream or the analysis results The contents of the resource object 32 are defined and 

produced from analyzing the event stream. Rather, the utilized during monitor 16 initialization. In addition, these 

resource object 32 provides a universally applicable syntax fields may be modified by internal monitor 16 components, 

for specifying the structure of event records and analysis and by authorized external clients using the monitor's 16 

results. Event records are defined based on the contents of an API. Modifying the resource object 32 permits adaptive 

event stream(s). Analysis result structures are used to pack- analysis of an event steeam, however, it also introduces a 

age the findings produced by analysis engines. Event records potential stability problem if dynamic modifications are not 

and analysis results are defined similariy to allow the tightly restricted to avoid cyclic modifications. To address 

eventual hierarchical processing of analysis results as event 45 this issue, monitors 16 can be configured to accept configu- 

records by subscriber monitors. ration requests from only higher-level monitors 16. 

Event-collection methods 36 gather and parse event Referring to FIG. 4, a monitor performs network surveil- 

records for analysis engine processing. Processing by analy- lance by monitoring 66 a stream of network packets. The 

sis engines is controlled by engine configuration 40a-40« monitor builds a statistical model of network activity from 

variables and data structures that specify the operating 50 the network packets, for example, by building 68 long-term 

configuration of a fielded monitor's analysis engine(s). The and short-term statistical profiles from measures derived 

resource object 32 maintains a separate collection of oper- from the network packets. The measures include measures 

ating parameters for each analysis engine instantiated in the that can show anomalous network activity characteristic of 

monitor 16. Analysis unit configuration 3Sa-^Hn include network inUusion such as measures that describe data 

configuration variables that define the semantics employed 55 transfers, network connections, privilege and network 

by the analysis engine to process the event stream. errors, and abnormal levels of network traffic. The monitor 

The resolver configuration 42 includes operating param- ^an compare 70 the long-term and short-tenn profiles to 

eters that specify the configuration of the resolver's internal delect suspicious network activity. Based on this 

modules. The decision unit configuration 44 describes companson, the monitor can respond 72 by reporting the 

semantics used by the resolver's decision unit for merging 60 activity to another monitor or by executing a countermea- 

the analysis results from the various analysis engines. The sure response. More information can be found m P. Porras 

semantics include the response criteria used to invoke coun- and A. Valdes ''Live Traffic Analysis of TCPAP Gateways", 

termeasure handlers. A resource object 32 may also include Networks and Distnbuted Systems Security Symposium, 

response methods 48. Response methods 48 include prepro- March 1998, which is incorporated by reference in its 

grammed countermeasure methods that the resolver may 65 entirety. 

invoke as event records are received. A response method 48 A few examples can illustrate this method of network 

includes evaluation metrics for determining the circum- surveillance. Network intrusion frequently causes large data 
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transfers, for example, when an intruder seeks to download 
sensitive files or replace system files with harmful substi- 
tutes. A statistical profile to delect anomalous data transfers 
might include a continuous measure of file transfer size, a 
categorical measure of the source or destination directory of 5 
the data transfer, and an intensity measure of commands 
corresponding to data transfers (e.g., commands that down- 
load data). These measures can detect a wide variety of data 
transfer techniques such as a large volume of small data 
transfers via e-mail or downloading large files en masse. The 
monitor may distinguish between network packets based on 
the time such packets were received by the network entity, 
permitting statistical analysis to distinguish between a nor- 
mal data transfer during a workday and an abnormal data 
transfer on a weekend evening. 

Attempted network intrusion may also produce anoma- 
lous levels of errors. For example, categorical and intensity 
measures derived from privilege errors may indicate 
attempts to access protected files, directories, or other net- 
work assets. Of course, privilege errors occur during normal 20 
network operation as users mistype commands or attempt to 
perform an operation unknowingly prohibited. By compar- 
ing the long-term and short-term statistical profiles, a moni- 
tor can distinguish between normal error levels and levels 
indicative of intrusion without burdening a network admin- 25 
istrator with the task of arbitrarily setting an unvarying 
threshold. Other measures based on errors, such as codes 
describing why a network entity rejected a network packet 
enable a monitor to detect attempts to infiltrate a network 
with suspicious packets. 3q 

Attempted network intrusion can also be detected by 
measures derived from network connection information. For 
example, a measure may be formed from the correlation 
(e.g., a ratio or a difference) of the number of SYN connec- 
tion request messages with the number of SYN_ACK 35 
connection acknowledgment messages and/or the number of 
ICMP messages sent. Generally, SYN requests received 
should balance with respect to the total of SYN^ACK and 
ICMP messages sent. That is, flow into and out-of a network 
entity should be conserved. An imbalance can indicate 4q 
repeated unsuccessful attempts to connect with a system, 
perhaps corresponding to a methodical search for an entry 
point to a system. Alternatively, intensity measures of 
transport-layer connection requests, such as a volume analy- 
sis of SYN-RST messages, could indicate the occurrence of 45 
a SYN-attack against port availability or possibly port- 
scanning. Variants of this can include intensity measures of 
TCP/FIN messages, considered a more stealthy form of port 
scanning. 

Many other measures can detect network intrusion. For 50 
example, "doorknob rattling," testing a variety of potentially 
valid commands to gain access (e.g., trying lo access a 
"system" account with a password of "system"), can be 
detected by a variety of categorical measures. A categorical 
measure of commands included in network packets can ss 
identify an unusual short-term set of commands indicative of 
"doorknob-rattling." Similarly, a categorical measure of 
protocol requests may also detect an unlikely mix of such 
requests. 

Measures of network packet volume can also help detect 60 
mahcious traffic, such as traffic intended to cause service 
denials or perform intelligence gathering, where such traffic 
may not necessarily be violating filtering policies. A mea- 
sure reflecting a sharp increase in the overall volume of 
discarded packets as well as a measure analyzing the dis- 65 
position of the discarded packets can provide insight into 
unintentionally malformed packets resulting from poor line 
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quality or internal errors in neighboring hosts. High volumes 
of discarded packets can also indicate more maliciously 
intended transmissions such as scarming of UPD ports or IP 
address scanning via ICMP echoes. Excessive number of 
mail expansion request commands (EXPN) may indicate 
intelligence gathering, for example, by spammers. 

A long-term and short-term statistical profile can be 
generated for each event stream. Thus, different event 
streams can "slice" network packet data in different ways. 
For example, an event stream may select only network 
packets having a source address corresponding to a satellite 
office. Thus, a long-term and short-term profile will be 
generated for the particular satellite office. Thus, although a 
satelhte office may have more privileges and should be 
expected to use more system resources than other external 
addresses, a profile of satellite office use can detect "address 
spoofing" (i.e., modifying packet information to have a 
source address of the satellite office). 

The same network packet event may produce records in 
more than one event stream. For example, one event stream 
may monitor packets for FTP commands while another 
event stream monitors packets from a particular address. In 
this case, an FTP command from the address would produce 
an event record in each stream. 

Referring to FIG. 5, a monitor may also "deinterleave." 
That is, the monitor may create and update 74, 76 more than 
one short-term profile for comparison 78 against a single 
long-term profile by identifying one of the multiple short- 
term profiles that will be updated by an event record in an 
event stream. For example, at any one time a network entity 
may handle several FTP "anonymous" sessions. If each 
network packet for all anonymous sessions were placed in a 
single short-term statistical profile, potentially intrusive 
activity of one anonymous session may be statistically 
ameliorated by non-intrusive sessions. By creating and 
updating short-term statistical profiles for each anonymous 
session, each anonymous session can be compared against 
the long-term profile of a normal FTP anonymous session. 
Deinterleaving can be done for a variety of sessions includ- 
ing HTTP sessions (e.g., a short-term profile for each 
browser session). 

Referring to FIG. 6, a computer platform 14 suitable for 
executing a network monitor 16 includes a display 50, a 
keyboard 54, a pointing device 58 such as a mouse, and a 
digital computer 56. The digital computer 56 includes 
memory 62, a processor 60, a mass storage device 64a, and 
other customary components such as a memory bus and 
peripheral bus. The platform 14 may further include a 
network connection 52. 

Mass storage device 64a can store instructions that form 
a monitor 16. Tlie instructions may be transferred to memory 
62 and processor 60 in the course of operation. The instruc- 
tions 16 can cause the display 50 to display images via an 
interface such as a graphical user interface. Of course, 
instructions may be stored on a variety of mass storage 
devices such as a floppy disk 64^, CD-ROM 64c, or PROM 
(not shown). 

Other embodiments are within the scope of the following 
claims. 
What is claimed is: 

1. A method of network surveillance, comprising: 
receiving network packets handled by a network entity; 
building at least one long-term and at least one short-term 
statistical profile from at least one measure of the 
network packets, the at least one measure monitoring 
data transfers, errors, or network connections; 
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comparing at least one long-term and at least one short- 
term statistical profile; and 

determining whether the difference between the short- 
term statistical profile and the long-term statistical 
profile indicates suspicious network activity. 

2. The method of claim 1, wherein the measure monitors 
data transfers by monitoring network packet data transfer 
commands. 

3. The method of claim 1, wherein the measure monitors 
data transfers by monitoring network packet data transfer 
errors. 

4. The method of claim 1, wherein the measure monitors 
data transfers by monitoring network packet data transfer 
volume. 

5. The method of claim 1, wherein the measure monitors 
network connections by monitoring network connection 
requests. 

6. The method of claim 1, wherein the measure monitors 
network connections by monitoring network connection 
denials. 

7. The method of claim 1, wherein the measure monitors 
network connections by monitoring a correlation of network 
connections requests and network connection denials. 

8. The method of claim 1, wherein the measure monitors 
errors by monitoring error codes included in a network 
packet. 

9. The method of claim 8, wherein an error code com- 
prises a privilege error code. 

10. The method of claim 8, wherein an error code com- 
prises an error code indicating a reason a packet was 
rejected. 

11. The method of claim 1, further comprising responding 
based on the determining whether the difference between the 
short-term statistical profile and the long-term statistical 
profile indicates suspicious network activity. 

12. The method of claim 11, wherein responding com- 
prises transmitting an event record to a network monitor. 

13. The method of claim 12, wherein transmitting the 
event record to a network monitor comprises transmitting 
the event record to a hierarchically higher network monitor. 

14. The method of claim 13, wherein transmitting the 
event record to a network monitor comprises transmitting 
the event record to a network monitor that receives event 
records from multiple network monitors. 

15. The method of claim 14, wherein the monitor that 
receives event records from multiple network monitors 
comprises a network monitor that correlates activity in the 
multiple network monitors based on the received event 
records, 

16. The method of claim 11, wherein responding com- 
prises altering analysis of the network packets. 

17. The method of claim 11, wherein responding com- 
prises severing a communication channel. 
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18. The method of claim 1, wherein the network packets 
comprise TCP/IP packets. 

19. The method of claim 1, wherein the network entity 
comprises a gateway, a router, or a proxy server. 

20. The method of claim 1, wherein the network entity 
comprises a virtual private network entity. 

21. A method of network surveillance, comprising: 
monitoring network packets handled by a network entity; 
building a long-term and multiple short-term statistical 

profiles of the network packets; 

comparing one of the multiple short-term statistical pro- 
files with the long-term statistical profile; and 

determining whether the difference between the one of the 
multiple short-term statistical profiles and the long- 
term statistical profile indicates suspicious network 
activity. 

22. The method of claim 21, wherein the multiple short- 
term statistical profiles comprise profiles that monitor dif- 
ferent anonymous FTP sessions. 

23. The method of claim 21, wherein building multiple 
short-term statistical profiles comprises deinterleaving pack- 
ets to identify a short-term statistical profile. 

24. A computer program product, disposed on a computer 
readable medium, the product including instructions for 
causing a processor to: 

receive network packets handled by a network entity; 

build at least one long-term and at least one short-term 
statistical profile from at least one measure of the 
network packets, the measure monitoring data 
transfers, errors, or network connections; 

compare at least one short-term and at least one long-term 
statistical profile; and 

determine whether the difference between the short- term 
statistical profile and the long-term statistical profile 
indicates suspicious network activity. 

25. A method of network surveillance, comprising: 
receiving packets at a virtual private network entity; and 
building at least one long-term and at least one short-term 

statistical profile based on the received packets, and 
comparing at least one long-term statistical profile with at 
least one short-term statistical profile to determine 
whether the packets indicate suspicious network activ- 
ity. 

26. The method of claim 25, further comprising decrypt- 
ing the packets before statistically analyzing the packets. 

27. The method of claim 25, further comprising not 
decrypting the packets before statistically analyzing the 
packets. 
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